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SUMWARY 


The  purpose  of  this  paper  Is  to  show  that  a  combination  of 
djmamlc  programming  and  the  classical  method  of  successive 
approximations  pemlts  a  systematic  study  of  va!'lou3  classes  of 
combinatorial  oroblems  arising  In  scheduling  th.eory,  comr;unl— 
cation  theory  and  network  theory. 

Although  the  method  cannot  guarantee  convergence  to  the 
actual  solution.  It  furnishes  a  monotonlc  sequence  of  approxi¬ 
mations  by  means  of  approximation  in  policy  space. 

An  Important  feature  of  the  method  Is  the  use  of  tl'.e 
solu  ton  of  sub— problems  of  considerable  magnitude  as  steps  In 
the  approximation  procedure.  With  the  aid  of  digital  computer’s 
and  the  techniques  of  dynamic  programming,  this  Is  a  feasible 
method . 

As  examples,  wo  discuss  the  R1 tchcock— Koopm  ins  transporta¬ 
tion  problem,  an  allocation  problem,  and  the  "travelling 
salesman"  problem. 
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DYNAMIC  PROORAMMING,  SUCCESSIVE  APPROXIMATIONS 
AND  VARIATIONAL  PROBLEMS  OP  COMBINATORIAL  NATURE 

Richard  Bellman 


^1 .  INTRODUCTION 

There  are  a  large  class  of  variational  problems  of  combi¬ 
natorial  nature  arising  from  various  studies  In  allocation, 
C3chedullng  and  communication  theory  which  offer  formidable 
analytic  and  computational  difficulties.  Although  many  of 
these  problems  are  theoretically  susceptll  le  to  the  functional 
equation  technique  of  dimamlc  programrr.lng ,  in  actuality  any 
direct  computational  solution  Is  blocked  by  the  large  number 
of  state  variables  occurring. 

In  this  paper  we  wish  to  Indicate  how  a  combination  of 
dynamic  pr*ogrammlng  and  successive  approximations,  wedded  for 
the  purpose  of  treating  nonlinear  control  processes^,  can  be 
used  to  provide  a  systematic.  If  not  complete,  procedure  for 
treating  various  questions  of  the  type  mentioned  above.  As  Is 
characteristic  of  approximation  In  policy  space,  we  will  obtain 
monotone  approxlrriatlon .  Many  new  prol  lems  arise  concerning 
convergence,  convergence  to  local  extrema,  rapidity  of 
convergence,  numerical  stability,  an!  so  forth. 

Wlt!i  the  aid  of  modeiTi  digital  computers,  we  employ  as  a 
single  step  In  i:.c  approximation  process  the  solution  of  sub- 
problems  which  were  once  difficult  In  their  own  right.  In  this 
paper  we  use  dynamic  progi'ammlng  to  accomplish  this;  In  our 
previous  paper^  we  utilized  dynamic  programming  and  the  solution 
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of  systems  of  linear  differential  systems.  Many  examples  can 
be  given  In  which  linear  programming,  and  a  blend  of  various 
techniques  can  be  utilized.  IVie  point  we  wish  to  emphasize  Is 
that  the  Introduction  of  the  digital  computer  affords  vast  new 
scope  to  that  general  factotum  of  analysis,  the  method  of 
successive  approximations. 

In  place  of  euiy  abstract  general  formulation  of  the 
approximation  procedures  that  can  be  used,  we  will  Illustrate 
the  basic  Ideas  by  means  of  some  examples. 

^2.  HITCHCOCK-KOOPMANS  TRANSPORTATION  PROBLEM 

Consider  a  set  of  sources,  possessing  quantities  of 

supplies,  Xj^,  and  a  set  of  demand  points,  T^,  with  re¬ 
quirements  rj  such  that  ^x^  •  Let  the  cost  of  shipping 

a  quantity  x^^^  from  to  be  a  given  function 

If  the  elegant  and  powerful  techniques  of 

2  *^4 

Dantzlg  ,  auid  more  recently  of  Ford  and  Fulkerson'"^  and  Prager 

resolve  the  problem  of  minimizing  the  total  cost,  5  64«(x,  .). 

1,J 

If,  however,  the  shipping  costs  are  nonlinear,  as,  for  example, 
occurs  when  fixed  charges  exist,  the  only  general  technique 
available  at  the  moment  appears  to  be  that  given  In  However, 
the  functional  technique  discussed  there,  combined  with  the 
technique  of  Lagrange  multipliers.  Is  only  feasible  computationally 
at  the  present  time  If  either  M  or  N  Is  small,  say  less 
than  or  equal  to  5.  On  the  other  hand.  If  one  Is  small,  the 
other  may  be  quite  large,  e.g.  1000  or  2000. 

Let  us  now  Indicate  how  the  method  of  successive  approxl— 
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matlons  permits  us  to  convert  the  general  transportation 
problem  Into  a  sequence  of  problems  which  may  be  readily 
resolved  via  dynamic  programming.  Since  we  are  using  approxi¬ 
mation  in  policy  space,  we  obtain  a  monotone  sequence  of 
decreasing  costs. 

To  begin  the  approximations,  let  the  supplies  at  the 
sources  S^,  S^,  . . . ,  be  allocated  in  an  arbitrary  fashion 
to  the  N  demand  points.  The  problem  of  shipping  the 
supplies  from  the  two  remaining  sources  and  to 

satisfy  the  remaining  demands  Is  now  solved  by  means  of 

c; 

functional  equations  as  described  in  As  indicated  there, 
the  solution  of  this  problem  Involves  only  sequences  of  functions 
of  one  variable,  and  is  very  simply  carried  out  with  the  aid  of 
a  digital  computer. 

The  next  step  of  the  approximation  procedure  consists  of 
using  the  distribution  of  supplies  from  S^,  . . . ,  as 

determined  in  this  way,  and  reallocating  the  supplies  shipped 
from  Sg  and  so  as  to  minimize  the  cost  of  shipping  frx)m 

these  two  sources. 

The  process  continues  in  this  way;  at  each  step  minimi¬ 
zing  over  supplies  shipped  from  Sj,  and  where 

^N+1  *  ^1* 

It  is  clear  that  the  sequence  of  costs  obtained  in  this 
way  is  monotone  decreasing.  However,  as  examples  show,  these 
costs  can  converge  to  a  relative  minimum  rather  than  an 

absolute  minimum.  Even  when  there  exists  only  one  mlnlBoai,  U) 
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absolute  minimum,  this  process,  as  an  example  constructed  by 
0.  Gross  shows,  can  converge  to  a  value  different  from  the 
mlnlrmioi  value.  We  shall  discuss  these  questions  In  more 
detail  subsequently. 

^3.  GENERAL  ALLOCATION  PROCESSES 

Consider  the  problem  of  determining  the  maximum  of 
N 

(1)  ^  •  •  •  *^1M^ 

over  all  satisfying  the  constraints 

lil  k  -  1,  2,  ...»  M,  x^j  >  0. 

As  we  know^,  the  problem  can  be  treated  by  means  of  recurrence 
relations  Involving  sequences  of  functions  of  M  variables. 

Let  us  Indicate  hovj  we  can  approach  the  solution  by  means  of 
sequences  of  functions  of  one  variable.  Consider  a  detennlna— 
tlon  of  the  quantities  1  «  1,  2,  ...,  N,  k  =  2,  . . . ,  M, 

satisfying  (2),  and  determine  the  x^^  so  as  to  maximize 
R„(x).  Tl'ils  Involves  sequences  of  functions  of  one  variable. 
Using  these  values  of  x^^^  and  the  previous  values  of  x^j^, 
k  -  3,  ...»  M,  determine  the  x^^^  so  as  to  maximize  Rj^(x). 
Continuing  In  this  way,  we  clearly  obtain  a  monotone  Increasing 
sequence  of  values  for  Rj^(x). 

S4.  A  NETWORK  PROBLEM 


As  a  final  application  of  these  approximation  techniques. 
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consldei'  the  problerf  of  tracing  a  path  through  N  points 
having  rr.lnlmum  "lengt^  . "  In  a  plane,  with  the  usual  metric, 
this  Is  the  travelling  salesman  problem.  Although  It  Is  not 
difficult  to  resolve  this  problem  by  means  of  functional 
equations  for  N  cf  small  magnitude,  say  N  <  10,  for  N  ■  48, 
the  direct  method  founders  on  dimensionality  difficulties. 

We  can,  however,  employ  successive  approximations  and  the 
feasibility  of  the  solution  for  small  N  to  obtain  an  approach 
to  the  problem  for  large  N.  Furthermore,  as  above,  these 
approximate  solutions  yield  monotone  decreasing  length. 

Le .  the  points  be  enumerated  In  some  fixed  order 
,  Pp,  ...,  Pj^.  Taking  the  first  10  points,  ( P^  ,P2, . . . ,  P^q)  , 
we  determine  via  functional  equations  a  path  of  minimum  length 
starting  at  P^^,  ending  at  P^^,  and  passing  through  the 

In  t  e  rme  dlate  points,  (P2,P^,«..,Pq).  Le  t  *^k2*^k5*  *  *  *  ^^10^ 
be  a  sequence  of  points  obtained  In  this  way.  Now  take  the 
points,  ^  ^k3' *  ■  ’  ^  ^10' 1  ^  repeat  this  procedure. 

Proceeding  In  this  fashion  we  obtain  a  sequence  Of  approximate 
solutions,  each  of  length  less  than  or  equal  to  that  of  the 
preceding . 

^5.  DISCUSSION 

It  Is  clear  that  there  are  a  large  number  ol'  variants  of 
the  procedures  discussed  In  the  foi’egolng  sections.  One  that 
seems  particularly  interesting  Is  that  of  using  a  random 
sequence  of  sets  Instead  of  the  sets  (l,?),  (2,3),  ..., 
described  In  ^2,  ancJ  proceeding  similarly  In  a  r«mdom  rather  than 
regular  fashion  In  the  pr-ocecsos  described  In  b3  and  b4. 
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